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(54) Multiple sampling via a time-indexed method to achieve wide dynamic ranges 



(57) An architecture for a digital pixel sensor (300) 
is disclosed in which the dynamic range of the sensor is 
increased by taking multiple samples of an image. In the 
preferred embodiment, an array of pixel elements is fab- 
ricated in an integrated circuit (302). Each of the pixel 
elements outputing a digital signal comprises a photo- 
detector integrated with an analog to digital converter. 
An array of threshold memory cells (304), each corre- 
sponding to one of the pixel elements, is also provided. 
An array of time memory cells (308), each correspond- 



ing to one of the pixel elements, establishes a different 
sampling time for each of the pixel elements for each of 
multiple samples. An array of data memory elements 
(310), each coupled to one of the pixel elements, is also 
provided. The data memory elements (31 0) are also fab- 
ricated in the integrated circuit to receive a value from 
a corresponding one of the pixel elements when the con- 
tent in a corresponding one of the threshold memory 
cells permits. As a result, multiple samples may be col- 
lected for an image to thereby extend the dynamic range 
of a photodetector. 
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Description 

[0001] The invention relates to image sensor sys- 
tems. More particularly, the invention relates to an im- 
age sensor architecture and associated method for fa- s 
cilitating image multiple sampling using a time-indexed 
approach to achieve a wide dynamic range. 
[0002] Digital photography is one of the most exciting 
technologies to have emerged during the twentieth cen- 
tury. With the appropriate hardware and software (and 10 
a little knowledge), anyone can put the principles of dig- 
ital photography to work. Digital cameras, for example, 
are on the cutting edge of digital photography. Recent 
product introductions, technological advancement, and 
price cuts, along with the emergence of email and the *s 
World Wide Web, have helped make the digital cameras 
one of the hottest new category of consumer electronics 
products. 

[0003] Digital cameras, however, do not work in the 
same way as traditional film cameras do. In fact, they 20 
are more closely related to computer scanners, copiers, 
or fax machines. Most digital cameras use an image 
sensor or photosensitive device, such as charged-cou- 
pled device (CCD) or complementary metal-oxide sem- 
iconductor (CMOS) to sense a scene. The photosensi- 25 
tive device reacts to light reflected from the scene and 
can translate the strength of that reaction into a numeric 
equivalent. By passing light through red, green, and blue 
filters, for example, the reaction can be gauged for each 
separate color spectrum. When the readings are com- 30 
bined and evaluated via software, the camera can de- 
termine the specific color of each element of the picture. 
Because the image is actually a collection of numeric 
data, it can easily be downloaded into a computer and 
manipulated for more artistic effects. 35 
[0004] Nevertheless, there are many cases in which 
digital cameras simply can not be used because of the 
limited resolution of the image sensors in today's digital 
cameras. Film-based photographs have immeasurably 
higher resolution than digital cameras. While traditional 40 
film-based technology typically has a resolution of tens 
millions of pixels, the image sensors in the digital cam- 
eras that could be produced at a price that is acceptable 
to consumers is slightly more than a million pixels today. 
[0005] Dynamic range is another critical figure of merit 45 
for image sensors used in digital cameras. The dynamic 
range of an image sensor is often not wide enough to 
capture scenes with both highlights and dark shadows. 
This is especially the case for CMOS sensors which, in 
general, have lower dynamic range than CCDs. so 
[0006] Previously suggested solutions for widening 
the dynamic range of these devices can be divided into 
three categories: 

• Compressing the response curve; 55 
Multiple sampling; and 
Control over integration time. 



[0007] The response curve is compressed by using a 
sensor that has a logarithmic response. 
[0008] There are two ways of doing this: 

The first approach is to use a CMOS sensor that 
operates in an instantaneous current read out 
mode. In this mode, the photocurrent generated by 
a photodetector is fed into a device that has a log- 
arithmic response, for example a diode connected 
MOS transistor, to compress the sensor transfer 
curve. Although this scheme can achieve very wide 
dynamic range, the resulting image quality is gen- 
erally poor due to a low signal-to-noise ratio (SNR). 

The second approach to compress the response 
curve uses a technique referred to as well capacity 
adjusting. Here, the dynamic range is enhanced by 
increasing well capacity one or more times during 
exposure time. During integration well capacity is 
monotonically increased to its maximum value. The 
excess photo-generated charge is drained via an 
overflow gate. This scheme, however, suffers from 
large fixed pattern noise and degradation in the 
SNR. 

[0009] In multiple sampling, a scene is imaged sever- 
al times at different exposure times and the data are 
combined to construct a high dynamic range image. For 
this approach to work at reasonable capture times, the 
read out process must be performed at speeds much 
higher than normal active pixel sensor (APS) speeds. 
The multiple sampling scheme effectively achieves a 
wide dynamic range. In reality, much data may need to 
be read out, which can be particularly burdensome for 
many types of image sensors. 

[0010] Controlling integration time is the third method 
that has some promising aspects in comparison with 
others. In essence, the exposure time of each pixel is 
individually adjusted so that they do not get saturated at 
the end of each integration period. There are many ways 
of achieving this. 

[001 1] One way is to place a set-reset flip-flop and an 
AND gate at each pixel to control the integration start 
time to achieve local exposure control. However, this ap- 
proach suffers the following limitations: 

* Each pixel is large due to the inclusion of the flip- 
flop and the AND gate. 

A large 'timestamp* memory is needed to store the 
exposure time of all pixels. The exposure time of 
each pixel can be determined by trying out various 
exposure times. When capturing a moving scene, 
the exposure times change so the 'timestamp' 
memory must be updated, which not only is burden- 
some but also causes image lag. 

Moreover, in addition to the column and row decod- 
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ers used for pixel read out, another column and row 
decoders are needed to control the flip-flops. 

[0012] A second way is known for an individual pixel 
reset (IPR) to achieve local exposure control, namely a 5 
second reset transistor is added to the standard three- 
transistor APS design so that the integration start time 
of each pixel can be controlled externally. The second 
way keeps the pixel size small but requires a large ex- 
ternal memory to store the exposure time for all of the 10 
pixels, and further requires memory refreshing and ad- 
ditional column and row decoders. Moreover, multiple 
reset pulses might need to be applied to each pixel 
throughout the reset period. The time control for reset- 
ting pulses could be quite complicated. is 
[0013] There is therefore a great need for a wide dy- 
namic range image sensor that overcomes some of the 
above shortcomings and, in particular, outputs image 
data having a wide dynamic range. Further, the sensor 
should not require an external timestamp memory and 
control logic to update the exposure times. 
[0014] The present invention thus provides an archi- 
tecture for a digital pixel sensor in which the dynamic 
range of the sensor is increased by taking samples of a 
subject to be recorded, where each sample is taken over 
an interval of a different duration than the other samples. 
The use of different recording intervals allows integra- 
tion of multiple photodetector signals relative to a 
threshold value and thus expands the dynamic range of 
the photodetector without saturating the picture ele- 
ments in the image. . 

[001 5] In a first aspect the present invention provides 
an image sensor, comprising: a sensor array that out- 
puts signals representing one or more images of a 
scene, wherein said sensor array is fabricated in an in- 
tegrated circuit; a data memory, in communication with 
said sensor array, for storing values representative of 
said signals; and a threshold memory, in communication 
with said sensor array, for storing threshold values for 
each picture element of said one or more images of a 
scene; wherein said data memory stores values for each 
picture element of said one or more images. 
[0016] In a second aspect the present invention pro- 
vides an image sensor, comprising: a sensor array that 
outputs signals representing one or more images of a 
scene, wherein said sensor array is fabricated in an in- 
tegrated circuit; a data memory, in communication with 
said sensor array, for storing values representative of 
said signals; and a time index memory, in communica- 
tion with said sensor array, for storing different time in- 
dex values for each of said multiple representations of 
said one or more images of a scene; wherein said data 
memory stores values for each picture element of said 
one or more images. 

[001 7] In a third aspect the present invention provides 
in an image sensor, comprising a sensor array that out- 
puts signals representing one or more images of a 
scene, wherein said sensor array is fabricated in an in- 



tegrated circuit; and a data memory, in communication 
with said sensor array, for storing vatues representative 
of said signals, an apparatus in which multiple represen- 
tations of said one or more images of a scene are col- 
lected and selectively stored as said values in said data 
memory to extend said sensor array dynamic range, 
said apparatus comprising: a threshold memory, in com- 
munication with said sensor array, for storing threshold 
values for each picture element of said one or more im- 
ages of a scene; and a time index memory, in commu- 
nication with said sensor array, for storing different time 
index values for each of said multiple representations of 
said one or more images of a scene, said time index 
memory providing supporting (weighted) information for 
properly combining frames of data to produce a final dig- 
ital image having expanded dynamic range; wherein 
said data memory stores values for each picture ele- 
ment of said one or more images. 
[0018] In a fourth aspect the present invention pro- 
vides a method for expanding the dynamic range of an 
image sensor, comprising: providing a sensor array that 
outputs signals representing one or more images of a 
scene, wherein said sensor array is fabricated in an in- 
tegrated circuit; and providing a data memory, in com- 
munication with said sensor array, for storing values rep- 
resentative of said signals; wherein multiple represen- 
tations of said one or more images of a scene are col- 
lected and selectively stored as said values in said data 
memory to extend said sensor array dynamic range. 
[0019] In a fifth aspect the present invention provides 
an image sensor, comprising: a sensor array including 
NxM digital pixels, each further including a photodetec- 
tor and a circuit, said photodetector producing a charge 
signal when said array is exposed to a scene, said circuit 
converting said charge signal to a digital signal, wherein 
said sensor array is fabricated in an integrated circuit; a 
data memory, in communication with said sensor array, 
having a plurality of data memory cells, each for storing 
said digital signal from one of the digital pixels; and a 
threshold memory, in communication with said sensor 
array, having N threshold memory cells, each for storing 
a threshold value for one of said digital pixels so that 
said digital signal in each of said data memory cells cor- 
responds to said threshold value in a corresponding one 
of said threshold memory cells. 

[0020] In the preferred embodiment of the invention, 
an array of pixel elements is fabricated in an integrated 
circuit. Each of the pixel elements outputs a digital signal 
and comprises a photodetector and an analog to digital 
converter. The photodetector is integrated with the an- 
alog to digital converter. 

[0021] An array of threshold memory cells, each cor- 
responding to one of the pixel elements, is also provid- 
ed. The threshold memory assures that a picture ele- 
ment which corresponds to a particular threshold mem- 
ory cell does not exceed a threshold value and, there- 
fore, does not provide a saturated signal for the picture 
element. Alternatively, the threshold memory assures 
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that a signal in a picture element which corresponds to 
a particular threshold memory cell is read out into a data 
memory cell only when the signal exceeds a value in the 
particular threshold memory cell. In essence, the thresh- 
old memory avoids the readout of unnecessary values 5 
to the memory cells (discussed below). 
[0022] An array of time memory cells, each corre- 
sponding to one of the pixel elements, establishes a dif- 
ferent exposure time for each of the pixel elements for 
each of multiple samples. 10 
[0023] An array of memory elements, each coupled 
to one of the pixel elements, is also provided. The mem- 
ory elements are also fabricated in the integrated circuit. 
The memory elements only receive a value from a cor- 
responding one of the pixel elements when the content 15 
in a corresponding one of the threshold memory cells 
permits. In this way, multiple samples may be collected 
for a subject to be recorded to thereby extend the dy- 
namic range of a photodetector. 

[0024] Integration of the photodetector and the mem- 20 
ory that implements this mechanism into the same inte- 
grated circuit avoids the latency that would be experi- 
enced if an external memory was used. 
[0025] An embodiment of the present invention will 
now be described, by way of example only, with refer- 25 
ence to the accompanying drawings, in which: 

Figure 1A is a block diagram that shows a CMOS 
image sensor or photosensitive chip in which the 
invention may be practiced; 30 

Figure 1B is a block diagram which shows a photo- 
diode modeled as a current source and a capacitor; 

Figure 2 is a block diagram which shows the archi- 35 
tecture of a digital pixel sensor, as described in U. 
S. Pat. No. 5,461,425; 

Figure 3 is a block diagram which shows an image 
sensor that includes a threshold memory, a time in- 40 
dex memory, and a separate data memory, where 
each of the memories and the digital pixel sensor 
are integrated into the same sensor according to the 
invention; 

45 

Figure 4 is a graph which shows an example of mul- 
tiple exposures; 

Figure 5A is a block diagram which shows a pair of 
exemplary threshold memory cells, time index so 
memory cells, and corresponding data memory 
cells according to the invention; and 



Figures 5B and 5C are graphs which show, respec- 
tively, two corresponding time integration process- 
es according to the invention. 

[0026] In the following detailed description of the in- 



55 



vention, some specific details are set forth to provide a 
thorough understanding of the presently preferred em- 
bodiment of the invention. However, it should be appar- 
ent to those skilled in the art that the invention may be 
practiced in embodiments that do not use the specific 
details set forth herein. Well known methods, proce- 
dures, components, and circuitry have not been de- 
scribed in detail to avoid unnecessarily obscuring the 
invention. 

[0027] In the following discussion, in references to the 
drawings like numerals refer to like parts throughout the 
several views. 

[0028] Figure 1A shows an image sensor or photo- 
sensitive chip 100 in which the invention may be prac- 
ticed. The image sensor 100 may be used in an image 
capturing device (e.g. a digital camera) for either sta- 
tionary or video photography, and produces digital im- 
age data. The photosensitive chip 1 00, which is typically 
fabricated on a substrate such as CMOS, comprises a 
plurality of photodetectors that are arranged in an array. 
For color applications, a mosaic of selectively transmis- 
sive filters is superimposed in registration with each of 
the photodetectors so that a first, second, and third se- 
lective group of photodetectors are made to sense three 
different color ranges, for example, the red, green, and 
blue range of the visible spectrum, respectively. The 
number of the photodetectors in the photosensitive chip 
100 typically determines the resolution of digital images 
resulting therefrom. The horizontal resolution is a func- 
tion of the number of photodetectors in a row 102, and 
the vertical resolution is a function of the number of pho- 
todetectors in a column 104. 

[0029] Each of the photodetectors comprises a pho- 
tosensor that produces an electronic signal when it is 
exposed to light. Generally, the photosensor is a photo- 
diode or a photogate in a CMOS sensor. Figure 1B 
shows a photodiode 120 that is modeled as a current 
source 122 and a capacitor 124. When a reset signal is 
applied at a Reset terminal 130, the capacitor 124 is fully 
charged by and nearly to Vcc through the transistor 1 28, 
at which point the photodiode 120 is ready for light inte- 
gration. 

[0030] As soon as the reset signal is dropped (i.e. the 
voltage level is changed), light integration starts. As 
more and more incident photons from light 126 strike 
the surface of the photodiode 120, the current of current 
source 122 increases. The capacitor 124 starts to dis- 
charge through the current source 122. Typically, the 
photodiode collects more photons for higher photon in- 
tensities and, as a result, the resistance of the resistor 
1 22 decreases. 

[0031] Consequently, a faster discharge signal Vout 
is produced. In other words, the signal from Vout is pro- 
portional to the incident photons which strike the photo- 
diode 120. This signal is alternatively referred to herein 
as an electronic signal or pixel charge signal. Optionally, 
a circuit 1 30 may be employed to enhance the electronic 
signal Vout to a desired level so that the output, i.e. the 
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pixel charge signal, is effectively coupled to following cir- 
cuitry. 

[0032] Operation of an image sensor comprises two 
processes: 

5 

The light integration process, as described above; 
and 

The read out process. 

10 

[0033] Each of these two processes is sustained for 
a controlled time interval. In the light integration proc- 
ess, each photodetector is initiated to accumulate inci- 
dent photons of the light and the accumulation is reflect- 
ed as a pixel charge signal. After the light integration is 
process, the photodetectors start the read out process 
during which the pixel charge signal in each photode- 
tector is read out via read out circuitry to a data bus or 
video bus. The interval during which the light integration 
process proceeds is referred to as exposure control or 20 
electronic shuttering, and it controls how much charge 
is accumulated by each of the photodiodes. 
[0034] Figure 2 duplicates Figure 1 of U.S. Pat. No. 
5,461,425 and shows that each photodetector 14 in- 
cludes an A/D converter in addition to a photosensor. 25 
Each of the photodetectors is referred to as a sensor 
pixel or a sensor element or digital pixel. This is done to 
indicate that the photodetector herein includes an ana- 
log-to-digital conversion circuit, as opposed to a photo- 
detector which is commonly seen in a conventional im- 30 
age sensor, and which includes a photosensor and pro- 
duces an analog signal. Further, the pixel element here- 
in is different from a conventional image sensor because 
it outputs digital signals that can be read out at a much 
higher speed than an analog signal can be read out in 35 
a conventional image sensor. Hence, the resultant im- 
age sensor is considered a digital pixel sensor (DPS). 
The preferred embodiment of the invention is based on 
such architecture in which a sensor element includes a 
photosensor and an analog-to-digital conversion circuit. 40 
[0035] The image sensor of Figure 2 is formed on a 
single integrated circuit chip 10. The image sensor core 
12 comprises a two-dimensional array of light detecting 
elements, each connected to a dedicated A/D converter 
which outputs a stream of bits representative of the an- 45 
alog output of the light detecting element. The combina- 
tion of a light detecting element and A/D converter con- 
stitutes a single pixel element 14. Each pixel element 
14 includes identical circuitry. Digital filters 16 on chip 
10 are connected to receive the digital streams from so 
each pixel element 14 and convert each digital stream 
to an eight-bit byte representative of one of 256 levels 
of light intensity detected by the respective pixel element 
14. 

[0036] In operation, an image is focused on the image 55 
sensor core 12 such that a different portion of the fo- 
cused image impinges on each pixel element 14. Each 
light detecting element comprises a phototransistor 



whose conductivity is related to the intensity of light im- 
pinging upon the base of the phototransistor. The analog 
current through the phototransistor thus corresponds to 
the intensity of light impinging upon the phototransistor. 
The analog signals from all phototransistors in the core 
12 are simultaneously converted into serial bit streams 
output from dedicated A/D converters clocked using a 
common clock driver 18. The serial bit streams, over a 
period of time, i.e. over a frame period, can then be proc- 
essed by filters 1 6 (on-chip or off-chip) to derive a signal 
representative of the intensity of light impinging on the 
phototransistor. 

[0037] After each clock cycle, one bit is latched at an 
output of each A/D converter within each pixel element 
14. To now transfer each bit generated by the pixel ele- 
ments 14 to the filters 16 after each clock cycle, each of 
the rows of pixel elements 14 are addressed in se- 
quence, using row decoder 20, until all rows of pixel el- 
ements 14 have been addressed. Upon addressing 
each row, the one-bit output of each pixel element 14 in 
the addressed row is coupled to a corresponding bit line 
22. The filters 16 process the bit stream from each pixel 
element 14 to generate an eight-bit value per pixel ele- 
ment 14 corresponding to the average intensity of light 
impinging on the respective pixel element 14 for that 
frame period. These eight-bit values may then be output 
from the chip 1 0, using a suitable multiplexer or shift reg- 
ister, and temporarily stored in a bit-mapped memory 
24. The memory 24 may then act as a frame buffer, 
where the light intensity values in memory 24 are se- 
quentially addressed for controlling the light output of 
corresponding pixels in a monitor. 
[0038] In a particular embodiment of Figure 2, assum- 
ing that sixty-four separate filters 16 are used for con- 
verting the bit streams output on sixty-four bit lines 22 
to eight-bit values. (A multiplexer at the output of the 
core 12 may reduce the number of required filters to, for 
example, sixteen.) The preferred interaction of filters 16 
with memory 24 is as follows. Immediately after a row 
of pixel elements 14 has been addressed, a control cir- 
cuit 26, using the address generated by row decoder 20, 
fetches a previous (or interim) eight-bit value stored in 
memory 24 for each pixel element 14 in the addressed 
row and loads this previous value into the proper one of 
the 64 filters 1 6 about to receive a new bit from that pixel 
element 14. Conventional memory addressing tech- 
niques and circuitry may be used for this process. The 
single bit output of the respective A/D converters in the 
addressed pixel elements 14 is then applied to a respec- 
tive one of the sixty-four filters 16 containing the previ- 
ous eight-bit value for that pixel element 14. Each filter 
16 then updates the previous eight-bit value with the 
new single bit of information to generate a new interim 
value. The now updated eight-bit value generated by 
each filter 16 is then transferred back into memory 24, 
under control of the control circuit 26. 
[0039] Referring to Figure 3, there is shown an image 
sensor 300 based on the digital pixel sensor according 
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to one embodiment of the invention. The digital pixel 
sensor 302 may be implemented according to U.S. Pat. 
No. 5,461,425 or U.S. Pat. No. 5,801,657, and outputs 
digital signals representing one or more images of a 
scene. A sense amplifier and latches 304 are coupled 
to the digital pixel sensor 302 to facilitate read out of the 
digital signals from the digital pixel sensor 302. Unlike 
the prior art, an image sensor 300 in accordance with 
the invention also includes a memory 304 (referred to 
herein as a threshold memory) for storing threshold val- 
ues, a memory 308 (referred to herein as a time index 
memory) for storing time index values, and a digital or 
data memory 310 that is large enough to accommodate 
a frame of image data from sensor 302. 
[0040] According to one embodiment of the invention, 
it is assumed that the sensor 302 is of N by M pixels and 
has k-bits. Thus, the size of the threshold memory 306 
is of N by M bits, and the size of the time index memory 
308 is of N by M by m bits, where m is the time resolution. 
The presently preferred pixel resolution of sensor 302 
is 1000 by 1000 in 10 bits. Thus, the threshold memory 
306 is a one-megabit memory, the time index memory 
308 is a two-megabit memory when the time index Is set 
to be T, 2T, 4T and 8T (i.e. two-bit resolution), and the 
digital memory 306 preferably has a size of at least 1 .2 
megabytes. 

[0041] As a result of the above memory configuration, 
each of the pixel elements in the sensor 302 can be 
stamped by each of the memory cells in the threshold 
memory 306, the time index memory 308, and the data 
memory 310. 

[0042] In operation, at each of the time indexes, for 
example, T, 2T, 4T and 8T, the sensor 302 is exposed 
to a target multiple (e.g. four) times, resulting in four im- 
ages at four different exposure times. Figure 4 shows 
an example of the multiple exposures. As shown in Fig- 
ure 4, frame 1 is created after time T, frame 2 is created 
after time 2T, frame 3 is created after time 4T, and frame 
4 is created after time 8T One of the advantages of hav- 
ing multiple images of the same target is the ability to 
expand the dynamic range of the image thus captured. 
Because of the relative short exposure time, frame 1 typ- 
ically captures information that is related to high illumi- 
nation areas in the target. Likewise, because of the rel- 
atively long exposure time, frame 4 typically captures 
information that is related to low illumination areas in the 
target. Frame 2 and frame 3 thus capture information 
that is related to gradually increased illumination areas 
in the target. As a result, the combination of the multiple 
images provides a very wide dynamic range. 
[0043] Generally, each of the frames is read out to a 
memory so that subsequent processing to combine the 
frames is possible. The architecture illustrated on Figure 
2 shows that the image data are read out to a memory 
24 through a plurality of lines (e.g. pins of sensor 10). 
Given the exposure times, the generated image data 
must be read out fast enough so that it does not affect 
the following frame. As is well known, the number of 



lines is limited to a practical packaging solution and of- 
ten far less than what is needed to accommodate the 
required speed. Therefore, the limited number of the 
lines becomes a bottleneck for data transmission from 
5 the sensors 14 to the memory 24. 

[0044] One of features in the invention is to place an 
on-chip memory in the image sensor, shown as the data 
memory 310 in Figure 3. Thus, there is no bottleneck for 
data transmission from the sensors 302 to the memory 
10 310. In operation, after one exposure time, a frame of 
data can be immediately read out to the memory 310. 
[0045] According to one aspect of the invention, after 
the first frame of data is read out into the memory 310, 
the second frame of data is selectively read out into the 
15 memory 310 to improve, update, or enhance the pixel 
values contained therein. Selection is controlled by the 
contents of the corresponding threshold memory 306. 
[0046] Figure 5A shows a pair of exemplary threshold 
memory cells 502 and 504, exemplary time index mem- 
20 ory cells 506 and 508, and exemplary corresponding da- 
ta memory cells 51 0 and 51 2. After a first exposure time 
T, as shown in Figure 5B, it is shown that the resultant 
signal 514 exceeds a predefined threshold V1. Thus, a 
flag, such as a binary value "1" which represents that 
25 the threshold value V1 has been exceeded, is stored in 
the cell 502, the exposure time T is stored in the cell 
506, and the resultant signal or a representation thereof 
(e.g. the value 240 in eight-bit precision) is stored in the 
cell 510. The value of threshold V1 is usually so deter- 
30 mined that further exposure to the photosensor that pro- 
duces resultant signal 514 can cause the photosensor 
to become saturated. Therefore, in view of the flag in 
the cell 502, there is no need to enhance the value 
stored in the cell 510 after the first exposure time T In 
35 reality, further update of the cell 510 could cause the 
loss of the data therein as it is now clear that the next 
value would be a saturated value. 
[0047] It is now assumed that a resultant signal 516 
produced by an adjacent photodetector is below the 
40 threshold V1 , as shown in Figure 5C. Therefore, the cell 
504 does not store the flag "1," assuming that the cell 
504 was reset to "0" when the operation starts. This per- 
mits the corresponding data ceil 512 to be updated or 
enhanced with new value that results from a next expo- 
45 sure. It should be noted that the exact contents to be 
stored in the cells 502, 504, 506, or 508 depend largely 
on an implementation preference. 
[0048] One of the key features of the invention is to 
provide a stamp on each of the photodetectors in the 
so sensor 302 or each of the data cells in the memory 31 0 
to prevent any saturated values from overwriting useful 
information in the memory 310. The contents in the time 
index memory are used individually so that the final im- 
age can be regenerated correctly. This allows the con- 
55 tents in the memory 310 to be updated properly after 
additional exposure times, or allows the frames of data 
to be combined properly, 

[0049] The advantages and benefits provided by the 
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image sensor 300 are numerous: 
[0050] One of the advantages is the elimination of the 
data transmission bottleneck presented in the architec- 
ture of U.S. Pat. No. 5,461,425. 

[0051] Secondly, the integration of the on-chip mem- 
ory 310 with the digital pixel sensor 302 does not affect 
the performance of the digital pixel sensor 302 but, rath- 
er, improves the overall performance of image sensor 
300 significantly. Such improvements include that of 
matched bandwidth when reading out the digital signals 
from the digital pixel sensor 302. 
[0052] Thirdly, a threshold memory is used to prevent 
the read out of unnecessary values to the data memory. 
[0053] In addition, the time index memory is used to 
provide supporting (weighted) information for properly 
combining the frames of data to produce a final digital 
image having expanded dynamic range. 
[0054] The architecture 200 is preferably implement- 
ed in a CMOS image sensor. The resultant image sen- 
sor may be advantageously employed in digital cameras 
that can provide superior or comparable image qualities 
as opposed to film image qualities. 
[0055] Although the invention is described herein with 
reference to the preferred embodiment, one skilled in 
the art will readily appreciate that other applications may 
be substituted for those set forth herein without depart- 
ing from the scope of the present invention set forth in 
the accompanying claims. 



Claims 

1 . An image sensor, comprising: 

a sensor array that outputs signals represent- 
ing one or more images of a scene, wherein 
said sensor array is fabricated in an integrated 

circuit; 

a data memory, in communication with said 
sensor array, for storing values representative 
of said signals; and 

a threshold memory, in communication with 
said sensor array, for storing threshold values 
for each picture element of said one or more 
images of a scene; 

wherein said data memory stores values for 
each picture element of said one or more imag- 
es. 

2. The image sensor of claim 1 , further comprising: 

a time index memory, in communication with said 
sensor array, for storing different time index values 
for each of said multiple representations of said one 
or more images of a scene. 

3. An image sensor, comprising: 

a sensor array that outputs signals represent- 



ing one or more images of a scene, wherein 
said sensor array is fabricated in an integrated 
circuit; 

a data memory, in communication with said 
5 sensor array, for storing values representative 

of said signals; and 

a time index memory, in communication with 
said sensor array, for storing different time in- 
dex values for each of said multiple represen- 
10 tations of said one or more images of a scene; 

wherein said data memory stores values for 
each picture element of said one or more imag- 
es. 

15 4. The image sensor of claims 1 and 2 or 3, wherein 
said sensor array is of N by M pixels and has k-bits. 

5. The image sensor of claims 1 , 2 and 4, wherein said 
threshold memory is of N by M bits. 

20 

6. The image sensor of claims 2 to 5, wherein said time 
index memory is of N by M by m bits, where m is a 
time resolution. 

25 7. The image sensor of claims 2 to 6 wherein, at each 
of said time indexes, said sensor array is exposed 
to a target multiple times, resulting in a plurality of 
images at a plurality of different exposure times. 

30 8. The image sensor of claim 7, wherein said plurality 
of different exposure times comprise any of a rela- 
tive short exposure time for capturing information 
that is related to high illumination areas in a target, 
a relatively long exposure time for capturing infor- 

35 mation that is related to low illumination areas in 
said target, and intermediate exposure times for 
capturing information that is related to gradually in- 
creased illumination areas in said target; 

wherein a combination of said muttiple imag- 

40 es establishes a wide dynamic range for said sen- 
sor array. 

9. The image sensor of claims 1 to 8, wherein each of 
multiple representations is read out to said data 

45 memory to combine said multiple representations 
for subsequent processing. 

10. The image sensor of claims 1 to 9 wherein, after a 
first frame of data is read out into said data memory, 

50 a second frame of data is selectively read out into 
said data memory to improve, update, or enhance 
pixel values contained therein. 

11. The image sensor of claim 10, wherein said selec- 
55 tive read out is controlled by the contents of said 

corresponding threshold memory. 

12. The image sensor of any one of the preceding 
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claims, wherein said image sensor is implemented 
as a CMOS image sensor. 

13. A method for expanding the dynamic range of an 
image sensor, comprising: 5 

providing a sensor array that outputs signals 
representing one or more images of a scene, 
wherein said sensor array is fabricated in an in- 
tegrated circuit; and 10 
providing a data memory, in communication 
with said sensor array, for storing values repre- 
sentative of said signals; 
wherein multiple representations of said one or 
more images of a scene are collected and se- 15 
lectively stored as said values in said data 
memory to extend said sensor array dynamic 
range. 

14. The method of claim 13, further comprising: 20 

providing a stamp on each detector in said 
sensor array or each data cell in said data memory 
to prevent any saturated values from overwriting 
useful information in said data memory. 

25 

15. The method of claims 13 or 14, further comprising: 

providing a time index memory in communica- 
tion with said sensor array; and 
using the contents in said time index memory 30 
individually to allow a final image to be regen- 
erated correctly; 

wherein the contents in said data memory are 
updated properly after additional exposure 
times. 35 

16. The method of claims 13 to 15, further comprising: 

providing a threshold memory for preventing 
read out of unnecessary values to said data mem- 
ory- 40 

17. In an image sensor, comprising a sensor array that 
outputs signals representing one or more images of 
a scene, wherein said sensor array is fabricated in 

an integrated circuit; and a data memory, in com- 45 
munication with said sensor array, for storing values 
representative of said signals, an apparatus in 
which multiple representations of said one or more 
images of a scene are collected and selectively 
stored as said values in said data memory to extend 50 
said sensor array dynamic range, said apparatus 
comprising: 



said sensor array, for storing different time in- 
dex values for each of said multiple represen- 
tations of said one or more images of a scene, 
said time index memory providing supporting 
(weighted) information for properly combining 
frames of data to produce a final digital image 
having expanded dynamic range; 
wherein said data memory stores values for 
each picture element of said one or more imag- 
es. 

18. An image sensor, comprising: 

a sensor array including NxM digital pixels, 
each further including a photodetector and a 
circuit, said photodetector producing a charge 
signal when said array is exposed to a scene, 
said circuit converting said charge signal to a 
digital signal, wherein said sensor array is fab- 
ricated in an integrated circuit; 
a data memory, in communication with said 
sensor array, having a plurality of data memory 
cells, each for storing said digital signal from 
one of the digital pixels; and 
a threshold memory, in communication with 
said sensor array, having N threshold memory 
cells, each for storing a threshold value for one 
of said digital pixels so that said digital signal in 
each of said data memory cells corresponds to 
said threshold value in a corresponding one of 
said threshold memory cells. 

19. The image sensor of claim 18, further comprising: 

a time index memory, in communication with 
said sensor array, for storing different time index 
values, each indexing said digital signal from one 
of the digital pixels. 

20. The image sensor of claims 18 or 19 wherein, after 
a first frame of data is read out into said data mem- 
ory, a second frame of data is selectively read out 
into said data memory to improve, update, or en- 
hance pixel values contained therein and wherein 
said first frame and said second frame are generat- 
ed in a same exposure to the scene. 

21. The image sensor of claim 20 wherein said digital 
signal from each of the digital pixels is readout only 
when said digital signal exceeds a predefined 
threshold. 



a threshold memory, in communication with 
said sensor array, for storing threshold values 55 
for each picture element of said one or more 
images of a scene; and 

a time index memory, in communication with 
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